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ABSTRACT 
Nonlinear  programming  methods  are  used  to  solve  saddle-point   prob- 
lems subject  to  inequality  constraints  on  the  variables;  in  particular,  the 
type  of  saddle-point  problem  arising  in  pursuit-evasion  differential  games 
is  considered.    The  methods  investigated  fall  into  two  groups:    solution 
of  the  nonlinear  simultaneous  equations  obtained  from  the  Kuhn-Tucker 
conditions,  and  solution  of  a  sequence  of  constrained  optimization  prob- 
lems by  the  gradient  projection  algorithm.    These  methods  are  applicable 
to  any  real-valued  function  f (x,yj  which  is  convex  in  x,  concave  in  y_,  and 
has  continuous  and  bounded  second  partial  derivatives.    Several  examples 
are  given  which  illustrate  the  characteristics  of  the  numerical  procedures  . 
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J  = 


[  (x  +  2t)2  +  xyt  -  (y  +  3t)2  ]  dt 


subject  to 


r»l 


x  dt  =  1    ,  x  :>  0 


y  dt  =  1    ,  y  £  0 . 


I.     INTRODUCTION 

The  original  development  of  differential  game  theory  is  credited  to 
Isaacs  for  his  work  during  the  195  0's   [8  ]  .     His  book  [9]  ,   published  in 
1965,   provides  a  comprehensive  summary  of  this  theory.     Issacs'  develop- 
ment closely  resembles  the  dynamic  programming  approach  to  optimal 
control  theory  popularized  by  Bellman  [2]  .     Many  subsequent  authors  have 
studied  differential  games;  among  the  significant  contributors  are  Ho, 
Bryson,  and  Baron  [6]  ,  and  Berkowitz  [3].     These  and  many  recent  papers, 
use  the  calculus  of  variations  as  an  analytical  tool. 

A  statement  of  the  differential  game  problem  is  as  follows:    A  dyna- 
mic system  is  described  by  the  state  equation 

x  =  f_(x(t),  u(t),  y_(t),  t)        ;       x(t0)=x0  (1) 

where  x  is  a  vector  which  describes  the  position,  or  state,    of  the  game, 
uis  a  control  vector  selected  by  player  1  and  v  is  a  control  vector  selec- 
ted by  player  2.     The  vectors  u_  and  v  are  members  of  the  sets  U  and  V 
respectively,  denoted  byucU    and  v  e  V,  where  U  and  V  are  to  be  speci- 
fied later.     The  performance  criterion  is  a  functional  assumed  to  be  of  the 
form 


J  =  h(x(t, ),  tf)  +    I    f   g(x(t),  u(t),  v(t),  t)dt, 
Jt 


(2) 


which  player  1  wishes  to  maximize,  and  player  2  wishes  to  minimize.    A 
saddle  point  of  J  is  sought  (if  it  exists) ,  i.e.  ,  a  u*€  U  and  v*cV    are 
sought  such  that 


J(u*v*)  £j(u*,  v*)  <  J(u*,  v)  (3) 

In  the  terminology  of  game  theory,  J  is  the  payoff,  u  and  v  are  strategies 
of  the  two  opposing  players,  and  u*  and  v*  are  optimal  pure  strategies. 
A  strategy  is  a  decision  rule  that  specifies  a  control  to  be  applied  by  a 
player,  based  on  all  information  available  to  him,  in  any  given  situation. 
Generally,  the  literature  has  considered  a  strategy  to  be  a  feedback  con- 
trol law,  that  is  ,  u  and  v  are  specified  in  terms  of  the  states  x;  however, 
Willman  [20]  has  used  the  concept  of  an  open-loop  strategy.     In  terms  of 
the  previous  definition,  this  means  that  the  information  available  to  each 
player  is  restricted  to  the  initial  state  and  time,  and  the  terminal  time. 
Note  that  u*  and  v*  are  minimax  strategies  in  the  sense  that  player  1 
maximizes  his  minimum  gain  while  player  2  minimizes  his  maximum  loss. 

The  concept  of  fixed  terminal    time  ignores  the  fact  that  in  more 
general  games  the  game  may  not  terminate  at  all.     Berkowitz  [4]  requires 
a  differential  game  to  terminate  whenever  t  and  x(t)  are  such  that 
(t,x(t))  is  a  point  of  a  previously  specified   set  in  (t,x)  space.    This  is 
the  "terminal  surface"  discussed  by  Isaacs  [9]  .     Berkowitz  [4]  also 
restricts  the  term  "differential  game"  to  include  only  two-person,   zero- 
sum  games  (the  type  described  above),  although  others  [19]   have  not 
been  so  restrictive  and  have  extended  the  theory  to  N-player,  non-zero- 
sum  games.     Zero-sum  refers  to  the  fact  that  there  is  a  single  performance 
criterion  which  one  player  tries  to  maximize  while  the  other  tries  to  min- 
imize.    Thus,  the  maximizing  player's  gain  is  the  minimizing  player's  loss 
and  the  sum  of  this  gain  and  loss  is  zero.     In  a  non-zero-sum  game,  the 
objectives  of  the  two  players  are  not  directly  opposed  --  each  may  have 


an  individual  performance  measure  to  minimize,  and  the  sum  of  the  two 
players'  criteria  is  not  necessarily  zero.    It  may  also  be  that  there  are 
more  than  two  players,  each  controlling  an  input  to  a  single  system,  and 
each  trying  to  minimize  his  individual  performance  criterion.     Such  prob- 
lems are  not  considered  in  this  thesis  . 

Although  variational  techniques  seem  to  be  an  appropriate  method 
for  solving  differential  games,  a  distinct  drawback  is  that  inequality 
constraints  on  the  value  of  the  state  and  control  variables  at  any  time 
are  difficult  to  include  in  the  analysis  [4]  .    Thus  ,  more  tractable  methods 
are  sought.    Since  the  problem  is  basically  an  optimization  problem  sub- 
ject to  inequality  constraints  it  is  natural  to  consider  mathematical  pro- 
gramming techniques  (nonlinear  programming  in  particular  since  the 
performance  measure  will  generally  be  nonlinear).    As  far  as  this  author 
can  determine,  only  one  other  paper  [10]   has  considered  this  approach. 

The  nonlinear  programming  problem  was  first  stated  mathematically 
by  Kuhn  and  Tucker  [14]  .    The  problem  is  to  find  an  x°    that  maximizes  a 
function  F(x)  subject  to  inequality  constraints  of  the  form  X^xJ^O1    anc*  2£ 
^0_.     Kuhn  and  Tucker  found  necessary  conditions  for  x°    to  be  a  solution 
to  this  problem.     If  F(x)  is  a  concave  function  in  the  region  where 
X(x)^0.  and  x  s-£,    the  conditions  are  also  sufficient.    Several  algorithms 
to  solve  this  problem  have  been  proposed  although  this  thesis  will  use 
only  Rosen's  gradient  projection  algorithm  [15]  . 


X(x) g  0    means  that  each  component  ui  the  vector  a{x/  is  non- 
negative  . 


The  use  of  nonlinear  programming  methods  to  solve  optimal  control 
problems  has  been  discussed  by  Rosen  [16]  ,   [17]  ,  and  by  Kirk  [12]  . 
This  thesis  will  discuss  the  connection  between  nonlinear  programming 
and  saddle-point  problems  such  as  arise  in  pursuit-evasion  differential 
games  ,  and  several  examples  which  illustrate  these  ideas  will  be  solved 
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II.     BASIC  PRINCIPLES 

Several  results  from  Game  Theory  will  be  required  and  are  stated 

here.    A  more  complete  discussion  of  these  theorems  and  their  proofs  can 

be  found  in  [11]   . 

Theorem  l_.    General  Min-Max  Theorem.    Let  f(x,  y_)  be  a  real-valued 

function  of  two  vectors  x  and  y_  which  lie  in  X  and  Y,  respectively,  where 

both  X  and  Y  are  closed,  bounded,  convex  sets.3     If  f  is  continuous, 

convex  in  x  for  each  y_,  and  concave  in  y_  for  each  x,3    then 

min  max  f(x,y)  =  max  min      __  f(x,y) 

xc  X  y_cY     ^~  ^  y_€  Y        xeX 

A  sufficient  condition  for  a  saddle  point  follows  from  the  next 

theorem . 

Theorem  2  .    A  necessary  and  sufficient  condition  for 

min  max  f(x,y_)=max  min      vf(x,_y_)     . 

is  that  there  exist  a  pair  x*e  X,  y_*e  Y     such  that  for  all  y_  in  Y  and  all 
x  in  X 


2  X    is  a  convex     set      if  the  straight  line  joining  any  two  points 
in  X  lies  entirely  within  X.    That  is  ,  if  x  1  ,    x  3   c  X  then 

x3     =  ex1    +  (l-6)x3      cX    for  all    0<e^l 

3  f  (x)  is  a  convex  function  if 

(l-6)f(x  °  )  +  SfCx1    )     *        f([l-6]x°     +    fix1    )  (i) 

for    0  ^  ©  <  1  and  for  all  x  °      and    x  1    .    The  negative  of  a  convex 
function  is  a  concave  function,  i.e.  ,  f(x)  is  a  concave  function  if  the 
sign  of  the  inequality  in  (i)  is  reversed. 
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f(x*,yj  £f(x*,£*)  ^f(x,^*)  .  (4) 

Thus,  a  sufficient  condition  for  the  existence  of  a  saddle  point  (in 

the  sense  of  (4)  above)  is  that  f  (x,  _yj  be   convex  in  x  for  all  y_  and  concave 

in  y_  for  all  x. 

Considering  each  side  of  the  double  inequality  in  (4)  separately, 

the  problem  can  be  viewed  as  two  optimization  problems  .     Looking  at  the 

right-hand  inequality  first,  and  specifying  the  set  Y  as  Y  =  (y  |  X      (y)  ^  0 , 

yi 

i=T  !...,&}  ,  the  problem 
max  f(x*,   yj 

where  y_  is  an  m-vector,  is  a  statement  of  a  Maximum  Problem  from  non- 
linear programming.     If  f(x,yj  has  continuous  first  partial  derivatives, 
the  Kuhn-Tucker  necessary  conditions  can  be  written.4     For  y_*  to  be  a 
solution  to  the  above  problem  it  is  necessary  that  y_*  and  the  vector  of 
Lagrange  multipliers    jU       satisfy  the  following  conditions 

i 

7    f(x*,z*)    +    Z    O    ,   7    X   ,  (y*)]    =    0  (5) 

y_  yi     y_    yi  — 


~Y 


X,,  (Z*)     My     =    °  (6) 


X      (v*)      £    0  (7) 

~y  ~ 


^y 


*    0  (8) 


4  The  original  problem  considered  by  Kuhn  and  Tucker  also  included 
a  nonnegativity  constraint  on  the  variables  ,    y_^£  ,  but  general  usage 
has  associated  the  term  Kuhn-Tucker  conditions  with  the  necessary  con- 
ditions for  this  problem  as  well  as  the  one  they  originally  considered. 
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where    7     f(x*,  y_*)    denotes  the  gradient  of  f(x*,  y)   with  respect  to  y_ 
evaluated  at  x* ,  Y_* ,  i.e., 


7     f(x*,  y_*) 


=    v* 


~df  (x*. 

*) 

t 

• 

5f  (x*, 

X) 

L      Sym 

_ 

y_  =  y_ 


and  that  the  constraint  qualification  [14]  be  satisfied.     The  constraint 
qualification  rules  out  pathological  behavior  on  the  boundary  of  the  con- 
straint set.     In  this  thesis,  only  linear  constraints  will  be  considered, 
hence  the  constraint  qualification  is  always  satisfied.     It  has  been  shown 
[14]  that  if  f(x*,  y_)  is  a  concave  function  of  y_,  then  the  conditions   (5)- 
(8)  are  also  sufficient. 

Similarly,  for  the  problem, 

minxcX     f^-'  Z^ 

subject  to    X  .  (x)  £  0        ,        i=l ,  2  ,  .  .  .  ,k 

.XI 

with  f  a  convex  differentiable  function  of  x  ,  an  n-vector,  the  necessary 
and  sufficient  conditions  are, 


-7     f(x*,y_*)  +  L      [jLi  .  7     X    .(x*)  ]  =   0 
2£  _t       X1     x     xi  —  — 


X     (x*)     y,       =   0 

— x    —        ^x 


X    (x*)        s    0 

— x  —  — 


u         ^    0 
r=-x  — 


(9) 

(10) 
(ID 
(12) 


where  u     is  a  vector  of  Lagrange    multipliers.    Thus,  for  (x*,  y_*)  to  be  a 
solution  of  the  constrained  saddle-point  problem,  equations  (5),   (6),   (9), 
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and  (10)  and  inequalities  (7),   (8),   (11),  and  (12)  must  be  satisfied. 
Although  the  Kuh.n~Tuck.er  conditions  require  only  that  the  function  be 
real-valued,  concave -convex,  and  differentiable  to  ensure  that  a  saddle 
point  exists,  the  methods  examined  in  this  thesis  will  also  require  the 
function  to  have  continuous  and  bounded  second  partial  derivatives  . 
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III.     NUMERICAL  METHODS 
The  methods  investigated  for  solving  the  constrained  saddle-point 
problem  may  be  divided  into  two  groups;    1)  solution  of  the  nonlinear 
simultaneous  equations  given  by  the  Kuhn-Tucker  conditions,  and    2) 
direct  solution  of  a  maximization  problem  and  a  minimization  problem. 
Methods  in  the  first  group  have  the  disadvantage  that  although  the  original 
problem  is  of  dimension  m+n,  the  Langrange  multipliers  must  also  be 
treated  as  variables  and  the  dimensionality  increases  to  m+n+k+1 . 

A.         SOLUTION  OF  NONLINEAR  SIMULTANEOUS  EQUATIONS 

1 .  Newton-Raphson  Method 

Jacob  and  Polak  [10]  suggest  a  generalized  Newton-Raphson 
method  for  solving  the  system  of  four(vector)  equations  (5),    (6),   (9),  and 
(10)  subject  to  the  four(vector)  inequalities  (7),    (8),   (11),  and  (12).    Con- 
vergence of  this  method  was  found  to  be  somewhat  sensitive  to  the 
starting  point,  although  convergence  was  quite  rapid  if  appropriate  start- 
ing points  were  chosen. 

2 .  Brown's  Algorithm5 

Brown  [5]  has  proposed  an  algorithm  to  solve  nonlinear  simul- 
taneous equations  which  requires  fewer  multiplications  than  the  Newton- 
Raphson  method.    It  is  quite  similar  to  the  Gauss-Seidel  process  for 
nonlinear  systems  of  equations.     Brown  has  observed  that  the  stability 


Subroutine  NLNSYS  at  NPS  Computer  Facility 
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and  convergence  of  the  algorithm  do  not  seem  to  be  dependent  on  the 
exact  evaluation  of  the  partials  and  first  differences  may  be  used  to 
approximate  the  partials  „    This  method  was  also  found  to  be  sensitive  to 
the  starting  point, 


This  method  suggested  itself  due  to  the  nonnegativity  con- 
straints on  the  multipliers  in  the  necessary  conditions.    There  are  several 


ways  of  formulating  this  problem;  the  one  chosen  was  to  minimize 

J  =  Ux(x)T  ^T    +   [MvjT  y^T 

subject  to 


(13) 


~7      f(x,yj    +    £   n  .    7      X    .(x)    =    0 
x       —  ^  ,     n    XI        x     xi  —  — 

—  1=1  — 

I 

7       f(x,yj    +    E    jl      7      X    .(yJ     =    0 

y_  i_1   yi    i  yi 


^(x) 

fe  o 

x  (z) 

— y 

*  o 

M 

£    0 

(14) 


—x  — 

u    •  £    0 

^y  - 

Although  the  gradient  projection  algorithm  can  be  altered  to  include 
equality  constraints  [15],  they  may  also  be  written  as  two  inequalities, 
i  .e . 


6  The  gradient  projection  algorithm  programmed  by  D .  E.  Kirk  was 
used  in  this  and  the  following  two  applications. 
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(15) 
0_    <    [   7     f(x,y)    +E.  u  ,    7      X   .(yj  ]  £  0  + 

—  z  x    y1     z    y1 

The  minimum  value  of  f  will  be  zero  and  the  values  of  x,  y_,  u    ,    U 
yielding  this  minimum  will  satisfy  the  Kuhn-Tucker  necessary  conditions 

B.         DIRECT  MINIMIZATION  AND  MAXIMIZATION  BY  GRADIENT 
PROJECTION 

1  .    Gradient  Projection  Solution  of  a  Sequence  of  Constrained 
Optimization  Problems 

Since  the  necessary  conditions  in  Rosen's  algorithm  are  equi- 
valent to  the  Kuhn-Tucker  conditions  (see  Appendix  A),  this  method  may 
be  described  in  terms  of  two  optimization  problems.    An  arbitrary  y_,  say 
y_0  ,  was  chosen  and  the  minimization  problem 

minx(Xf(x,Xo)  (16) 

subject  to 

Xx(x)  £0        , 

was  solved  for  Xj..    Then  a  solution  y_x  was  found  for  the  maximization 
problem 

mdXy_eY  f(2il  '^  (17) 

subject  to 

This  procedure  was  continued  until  some  stopping  criterion  was  met.    A 
convenient  stopping  criterion  is 
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If  this  criterion  is  met,  the  point  (xt  ,  ,y_i)  found  is  the  desired  solution 
by  the  definition  of  a  saddle  point    -  -  equation  (4) .    There  remain  to  be 
shown  the  conditions  under  which  the  sequence  of  points  (xx  ,x2  ,  .  .  .) 
and  {y_0,£i  ,  .  . .)   converge  to  x*  and  y_*.    The  Principle  of  Contraction 
Mapping  [13]  was  applied  to  the  class  of  functions  considered  in  this 
thesis  and  conditions  were  found  which  ensure  convergence  for  these 
functions   (see  Appendix  B) . 

2  .    Simultaneous  Minimaximization  by  Gradient  Projection 

Before  describing  this  method  it  should  be  stated  that  it  does 
not  seem  to  work  in  general;  it  is  included  primarily  to  point  out  some  of 
the  pitfalls  of  using  two  dimensions  to  visualize  n-dimensional  concepts 

The  problem  of  finding  a  saddle   point  of  f(x,y_)  subject  to 

X    (x)    2:  0   and  X    (y)  ^  0   was  altered  in  the  following  manner.     Let 
— x  —  ~y  ~ 


x 


X 


X    (z) 

— z 


X   (x) 

— X 


X   I 


Define  the  "gradient"  of  f(z)  as 


(18) 


It  was  then  intended  to  solve  the  Maximum  Problem 


max     f(z) 


subject  to 


X     (z)    *  0 

~~  z  — 


with  the  "gradient"  as  defined  in  equation  (18),  for 


*  ■  ES 


(19) 


For  the  case  where  x  and  y_  are  scalars,  the  progress  of  the  algorithm  may 
be  followed  on  a  contour  plot  and  converges  quite  rapidly.     Unfortunately 
this  method  does  not  seem  to  extend  to  n-dimensions  except  for  isolated 
examples. 
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IV.     TWO-DIMENSIONAL  EXAMPLES 
The  methods  described  in  the  preceding  section  were  initially 

tested  on  the  following  two-dimensional  example.    Find  the  saddle 

point  of 

f(x,y)  =  x3  -   y3  +  xy  +  2x  +  4y   -    6  (20) 

subject  to 

M"   ^  x   <  M+ 
x  x 

(21) 

M~    <   y   <M+ 

y  y 

The  function  f  is  convex  in  x  and  concave  in  y,  hence,  by  Theorems  1 
and  2  a  saddle   point  exists.    The  region  of  x,y  space  for  which  the 
inequalities  (21)  are  satisfied  is  called  the  feasible  region.    If  this 
feasible  region  is  large  enough,  the  saddle    point  will  be  interior  and 
located  at 

x*   =  -1.6 

y*  =     1.2 

Clearly,  the  selection  of  M+  ,  M"  ,  M+  ,  and  M"  in    (21)  determines 

x        x         y  y 

whether  the  solution  will  be  interior  or  constrained. 

From  equations  (5)  through  (12)  necessary  conditions  for  (x*,y*) 
to  be  a  saddle  point  are 
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(22) 


(23) 


-2x*    -2    -   y*    +  Mxl  -   »x2       =    0 

(x*   +  M"x)  Mxl  =    0 

(-x*  +  M+  )  u   _  =    0 
x       x2 

-2y*  -4+x*+un-/Lt0        =0 
yi  y2 

(y*+M")  |iyl    =    0 
(-y*  +  M+y)My2     =    0 

x*  +  M"     ^  0  a    ,  Jfc   0 

x  xl 

-x*   +  M+     ^  0  ju    _    *  0 

x  x2 

y*    +   M~     £  0  U    .     ^   0 

y  yi 

-y*    +   M+     ^  0  h-    _     s   0 

y  y2 

A.         SOLUTION  OF  NONLINEAR  SIMULTANEOUS  EQUATIONS 

All  three  methods  converged  for  the  several  cases  of  the  previous 

example  which  were  examined. 

With  M  =  M+     =     M~     =    M+     =    M"     =  3 .  ,  the  three  methods 
x  x  y  y 

converged  from  the  starting  point  (1,-1,1,1,1,1)  to 

x*      =   -1.6000 

y*      =      1.2000 

"xl'    Mx2'    "yl'    ^y2      =      °;° 

The  Newton-Raphson  method  required  four  iterations,  Brown's  algorithm 
took  three  iterations,  and  the  gradient  projection  root-finding  method 
required  five  iterations.    These  results  are  shown  in  Figs.   1,2,  and  3 
respectively. 

21 


1.000  -1.000 

-0.781  0.898 

-1.526  1.202 

-1.597  1.201 

-1.600  1.200 


Fig.   1 
Contours  of  f(x/y)  =  x2  -  y2  +  xy  +  2x  +  4y   -    6    showing 
progress  of  the  Newton-Raphson  method  to  an  interior  saddle 
point . 
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1  -1.384        1.238 

2  -1.582        1.206 

3  -1.600        1.200 


Fig.  2 
Contours  of  f(x,y)    =  xa  -   y2  +  xy  +  2x  +  4y   -    6  showing  progress 
Brown's  Algorithm  to  an  interior  saddle  point. 
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Fig.  3 
Contours  of  f(x,y)    =  x3  -  y2  +  xy  +  2x  +  4y   -   6    showing  progress  of 
gradient  projection  root-finding  method  to  an  interior  saddle  point. 
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M+     was     changed  to  1 . 0  and  all  three  methods  converged  to  the 

y 

constrained  saddle  point 

x*      =   -1.5000 
y*      =      1.0000 

"xl'    ^x2'    ^yl      =      °'° 

U    _      =      0.5000 
y2 

in  five,  four,  and  six  iterations  respectively.     These  results  are  shown 

in  Figs.  4,5,  and  6.     Note  that  a  nonfeasible  starting  point  was  used 

for  the  gradient  projection  root-finding  method  in  Fig.   6.    Also  note 

that  the  method  does  not  seem  to  make  any  progress  from  iteration  1  to 

iteration  2  .    This  is  due  to  the  fact  that  only  two  of  the  six  variables  are 

shown  in  Fig  .   6  . 

The  objective  function  was  then  changed  slightly  (for  reasons  to 
be  explained  in  a  following  section)  to 

f(x,y)    =  x2   -    y2   +   5xy   +      2x   +   4y    -    6 
With  M   =   3 .  ,  all  three  methods  converged  to  the  interior  saddle-point 

x*      =   -0.82759      _- 
y*      =   -0.06897 

Mxl'    ^x2'    "yl'    ^y2      =      °'° 

B.         DIRECT  MINIMIZATION  AND  MAXIMIZATION  BY  GRADIENT 
PROJECTION 

1 .    Gradient  Projection  Solution  of  a  Sequence  of  Constrained 
Constrained  Optimization  Problems 

The  function 

f(x,y)    =  x2   -    y2   +  xy   +   2x   +   4y    -    6  (24) 
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Fig.  4 
Contours  of  f(x,y)   =  x2  -   y2  +  xy  +  2x  +  4y   -   6    showing  progress  of 
the  Newton-Raphson  method  to  a  constrained  saddle  point. 
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0 

1.000 

-1.000 

1 

-1.288 

0.984 

2 

-1.468 

0.967 

3 

-1.497 

0.995 

4 

-1.500 

1.000 

Fig.   5 
Contours  of  f(x,y)   =  xa  -   ya  +  xy  +  2x  +  4y   -   6    showing  progress  of 
Brown's  Algorithm  to  a  constrained  saddle  point. 
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0 

3.000 

3.000 

1 

-0.286 

1.000 

2 

-0.287 

0.999 

3 

-0.963 

-0.225 

4 

-0.659 

0.191 

5 

-1.228 

0.456 

6 

-1.500 

1.000 

Fig.  6 
Contours  of  f(x,y)  =  xa  -  y2  +  xy  +  2x  +  4y   -   6    showing  progress  of 
gradient  projection  root-finding  method  to  a  constrained  saddle  point. 
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was  again  used.    A  starting  value  of  y  was  selected  as  3,   so  the  first 
stage  of  the  method  was  to  minimize 

f(x)    =  x3  +   5x   -    3  (25) 

subject  to 

-   3    <  x   <  3      . 
A  solution  of  x1   =  -2.5   was  obtained. 
Then,  the  maximization  problem 


max     f(y)    =   -y2  +    1.5y   -   4.75 


(26) 


subject  to 


-   3    <    y   <    3, 
was  solved  and  the  solution  y2  =  0.75  was  obtained.     This  procedure  was 
continued  until  the  stopping  criterion  was  less  than  10"6.    The  progress 
of  the  algorithm  is  shown  in  Fig.   7.    The  sequence  of  points  (xl7 

x2/ )  and  (yx ,    y2, )  obviously  converges  to  x*  and  y*. 

Again,  the  objective  function  was  change  to 

f(x,y)    =  x2   -    y2   +   5xy   +   2x   +   4y   -    6         .  (27) 

This  time  the  sequence  of  points  (Xjjj)  (starting  from  the  origin)  diverged 
as  shown  in  Fig.  8.    A  sufficient  condition  for  convergence  is  that  the 


equation 


—  l   —      —  i  -i 


(2  8) 


be  a  contraction  mapping  (see  Appendix  B) ,  where 


z.1    = 


x, 


Lyd 


c  = 


-I 


25/4  0 

0  25/4 


Neither  of  the  sufficient  conditions  for  C  to  be  a  contraction  mapping  are 
satisfied  since 
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Fig.  7 
Contours  of  f(x,y)   =  xs  -  y2  +  xy  +  2x  +  4y   -   6    showing  progress  of 
sequential  min-max  by  gradient  projection  to  an  interior  saddle  point. 
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(es) 


i  < 


oesi 

a! 


C""7V        \T 


R'    £5_>>   i 


{S(tifi)t2    .3 


Fig.   8 
Contours  of  f(x,y)   =  xa  -   y2  +  5xy  +  2x  +  4y  -   6    showing  progress  of 
sequential  min-max  by  gradient  projection. 
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{i,  r,^)2}   = 


25  \2 

tj    + 


25  \ 


2     I    - 


1250 
16 


>  1 


(2  9) 


and 


Xi 


625 
16 


>  1 


(30) 


Furthermore,  a  sufficient  condition  for  C  not  to  be  a  contraction  mapping 


is  satisfied ,  i.e., 
625 


^ 


N 


16 


>  1 


Note  that  these  conditions  are  independent  of  the  initial  values  of  x  and 

J 


-For  the  previous  case  where  the  coefficient  of  the  xy  term,  was 


1  ,    the  conditions  for  C  to  be  a  contraction  mapping  are  satisfied,  since 


t-. 


0 


{I.  Sjfcij)1}^    W  +  (i>2}  =:8    <l 


and 


*i   = 


Te    <> 


2  .    Simultaneous  Minimaximization  by  Gradient  Projection 
Again ,  the  function 
22sr^f(x,  y)    =  x2   -   ys  +  xy   +   2x  +  4y   -    6  (31) 

subject  to 

-  3    <  x   ^    3, 

-  3    £  y   <;    3o 

is  used  to  illustrated  the  method. 

The  starting  point  x0  =  1 ,    y0  =   i   was  selected,  and  the  nega- 
tive gradient  with  respect  to  x  and  the  positive  gradient  with  respect  to  y 
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were  computed.    These  two  components  form  the  "plus -minus  gradient" 
defined  in  equation  (18).     They  are  shown  by  heavy  lines  in  Fig.   9.    The 
true  gradient  and  it's  components  are  shown  by  dotted  lines.    A  step  is 
taken  in  this   "plus-minus"  gradient  direction.    The  gradient  projection 
algorithm  takes  the  maximum  step  in  the  direction  of  the  gradient  (in  this 
case  the  plus-minus  gradient),  i.e.  ,  it  moves  as  far  as  possible  without 
leaving  the  bounded  search  region.     In  this  case  the  point    1      (-1.6,3.), 
which  lies  on  the  constraint  boundary,  is  reached.     The  algorithm  then 
determines  whether  or  not  this  maximum  step  is  too  far  by  computing  the 
inner  product  of  the  normalized  projected  gradient  at  point  0  and  the 
gradient  at  point  1'.    If  this  inner  product  is  greater  than  or  equal  to  zero 
the  maximum  step  is  taken.     If  it  is  negative,  repeated  linear  interpolation 
is  used  to  determine  the  point  where  the  inner  product  is  zero,  i.e.  , 
where  the  gradient  is  orthogonal  to  the  normalized  projected  gradient 
evaluated  at  the  previous  point.    In  this  case,  the  maximum  step  is  too 
far  and  by  repeated  linear  interpolation  the  point  1  is  obtained.    This 
procedure  was  continued  until  the  necessary  conditions  for  convergence 
of  the  algorithm  were  met. 
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Fig.  9 
Contours  of  f(x,y)   =  x2  -  y2  +  xy  +  2x  +  4y  -   6    showing  progress  of 
simultaneous  min-max  by  gradient  projection  to  an  interior  saddle  point. 
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V.     A  SPECIAL  CASE 
In  his  investigation  of  two-sided  allocation  problems,  Bartley  [1  ] 
proposed  a  solution  for  a  special  type  of  saddle-point  problem. 
A  functional  of  the  form 


I  = 


t> 


K  dt  (32) 


is  considered  where  K  =  K(x,  y,  t)  is  a  real  valued  function.    It  is  desired 
to  find  a  function  x  which  minimizes   (32)  subject  to  the  constraints 

x   £    0  and  X  =        x  dt   =    1  (33) 


b 

y    £    0  and  Y  =  y  dt   =    1  (34) 

a 


and  a  function  y  which  maximizes  (13)    subject  to 

J 

i.e.,  a  saddle    point  of  the  functional  I  is  sought  subject  to  the  constraints 
(33)  and  (34). 

It  is  convenient  to  introduce  a  new  functional 

„  b  „  b 

J  =  H  dt   =        i      (D  +  ji  x  +  /Li  y)    dt  (35) 

"a  "  a  x  y 

where   a      and  u     are  constants  (Lagrange  multiplier's)  to  be  found.     Note 
•x  *y 

that  if  x  and  y  are  admissible  (equations   (33)  and  (34)  are  satisfied)  then 

J  differs  from  I  by  the  amount   a     +  u    . 

x  y 

Now  if  the  functions  x°  and  y°  satisfy 

J(x°7   y)    <    J(x°,    y°)    <    J(x,   y°)  (36) 

they  are  a  saddle-point  of  J.    But  they  may  not  be  admissible  functions  . 
If  they  are  admissible  and  satisfy  (36)  they  are  called  the  min-max 
strategies,  x*  and  y* . 
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Bartley  proposed  that   M     and   u     be  selected  arbitrarily,    Then  the 

x  y 

Kuhn-Tucker  necessary  conditions  for  a  saddle-point  [14]  can  be  applied 

to  the  integrand,  H,  of  (35).     These  conditions  are 


1H   (x°,   y°)    ^   0     ,    rSJL(x°,    y0)-  • 
Sx  Ld  x 


iH_(x°,    y°)    *    0      ,    ("AH    (x°,    y0^- 
by  Ld  y 


xu  =   0    ,    xu  ;>    0 

(37) 


[' 

for  all  tc  [a,b]  ,  can  be  used  to  obtain  a  trial  solution  (x°    ,    y°).     If  this 

trial  solution  is  admissible,  then  the  min-max  strategies  have  been  found 

If  the  trial  solution  is  not  admissible,    Bartley  used  a  specially  tailored 

Newton-Raphson  iteration  to  correct  the  value  of  (X,  Y)  and  obtain  a 

sequence  approaching  (1,1). 

A  particular  functional  considered  by  Bartley  was, 

r1 
J(x,  y)      =      ,    [(x+2t)§  +xyt   -    (y  +   3t)s    ]   dt  (38) 

Jo 

By  dividing  the  interval  [0 ,  1  ]  into  N  subintervals  each  of  length  T  and 

assuming  piecewise-constant  x  and  y  over  T,  the  integrals  can  be 

approximated  by  summations.     (Note  the  if  the  problem  is  to  be  solved  on 

the  computer  there  is  a  discretization  performed  eventually  anyway) . 

Now  the  problem  is  in  the  form:    Find  a  saddle    point  of 

N-l 

2 

k=0 


I      ==  T  E        [  (x(k)  +  2  ^)2  +  x(k)y(k)-~    -  (y(k)  +  3^-)a  ]    (39) 


subject  to 

x(k)    £    0  ,  k   =  0,  .  .  .  .  ,N-1 

(40) 
y(k)    ^     0  ,  k   =   0, ,N-1 
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and 

N-l 
T   S     x(k)    =    1 

k=0 

(41) 

N-l 
T  L     y(k)   =   1 

k=0 

where  the  saddle  point  of  J     ,  which  is  a  function  of  2N  variables  subject 
to  2N+4  constraints  (if  the  equalities  are    written  as  two  inequalities  as 
in  equation  (15)  ),  can  be  found  by  the  methods  described  previously. 

This  example  was  solved  by  the  sequential  min-max  gradient 
projection  method  with  N=25  and  gave  the  trajectories  shown  in  Fig.   10. 
These  correspond  quite  closely  to  Bartley's  results,  also  shown  in  Fig. 
10. 

The  simultaneous  min-max  method  was  also  tried  on  this  problem 
and  gave  the  same  results  as  the  sequential  method.     Although,  as 
mentioned  previously,  the  simultaneous  min-max  method  does  not  seem 
to  extend  in  general  to  examples  of  dimension  greater  than  second-order, 
the  feasible  region  in  this  case  was  apparently  so  restricted  by  the 
equalities  that  the  algorithm  was  able  to  converge. 
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Sequential  Min-Max  by 
Gradient  Projection 

Bartley's  results 


Functions  x  and  y  which  are  a  saddle  point  of 

o  1 


J  = 


subject  to 


[  (x  +  2t)2  +  xyt  -    (y  +  3t)3  ]  dt 


J1xdt  =  l/    x   ^  0 ,       [1ydt=l/y^0 


38 


VI.    A  PURSUIT-EVASION  PROBLEM 
A  pursuit-evasion  problem  was  also  considered  in  which  each 
player's  dynamic  system  is  described  by  linear  differential  equations  and 
the  payoff  is  a  weighted  combination  of  terminal  miss  distance  and  each 
player's  control  effort  expended. 

The  state  equations  of  two  systems  are 

xp(t)    =Ap(t)xp(t)    +Bp(t)up(t)    ;xp(t0)    =xp() 

xjt)    =  Ae(t)xe(t)   +   Be(t)ue(t)    ;   xe(tQ)    =  XeQ  ^ 

u    (t),    u    (t)    c  Eq 

-p  ~e 

where  x     and  x     are  n-vectors  describing  the  respective  players'  states 
— p  —e 

at  any  time  t,  u     and  u     are  q-dimensional  control  vectors  for  each 
-p  -e 

7  Q 

player,  and  E     is  q-dimensional  Euclidean  space.    The  problem  is  to 

find  a  saddle  point  (u   *,  u   *)  of  the  payoff 

P  e 

T=    6Cp(tf)    -x^)]1      QLxp(tf)    -xe(tf)    ] 

(43) 

**  [u    (t)T   R      u    (t)     -   u    (t)T   R   u    (t)]     dt 
t        — p  p  —  p  -e  e—e 

0 

where  Q  is  a  positive  semidefinite  matrix  and  R     and  R     are  positive 

P  e 

definite  matrices.    This  is  essentially  the  problem  solved  by  Ho,  Bryson, 

and  Baron  [6]  .    The  solution  is  well  known  and  has  been  presented  by 
many  authors,  including  Hutchison  and  Permenter    [7],  and  Willman  [20]. 


7      The  dimension  of  each  player's  control  vector  may  be  different 
although  for  simplicity  they  are  assumed  equal  here. 
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Additional  constraints  on  the  control  variables  are  assumed,  such 
as 

|u    ,{t)  |   <    M         ,  i  =   1,..'.  ,q  (44) 

pi  p 

|ue.{t)|   <    Mg       ,  i  =   l,...,q  .  (45) 

Any  other  equality  or  inequality  constraints  on  either  the  state  or  control 
variables  may  also  be  included  (Jacob  and  Polak  [10]  include  energy 
constraints),  but  for  simplicity  only  the  above  are  considered.    The  cal- 
culus of  variations  as  an  analytical  tool  is  no  longer  as  attractive  as  it 
was  in  the  unconstrained  problem.    Although  Berkowitz,  in  his  definitive 
paper  based  on  the  calculus  of  variations  [3]   ,  considered  constraints 
theoretically,  apparently  variational  techniques  have  not  been  applied 
to  solve  meaningful  examples  with  constraints.    This  deficiency  is 
important  because  it  is  quite  realistic  to  expect  constraints  on  the  con- 
trol  variables  of  physical  systems,  e.g.  ,  aircraft  do  not  have  unlimited 
control  surface  deflection  nor  unlimited  acceleration.    Furthermore,  in  a 
pursuit-evasion  situation  it  is  realistic  to  expect  control  variables  to 
always  be  at  their  limits  ,  e.g.  ,  an  aircraft  commander  will  use  every 
bit  of  maneuverability  built  into  his  aircraft  to  avoid  a  pursuing  missile. 

The  procedure  used  to  solve  this  problem  was  to  assume  that  the 
control  variables  are  piecewise  constant,  and  to  approximate  the  differ- 
ential equations  by  difference  equations  and  the  integral  in  the  payoff 
functional  by  a  summation.    Then  the  nonlinear  programming  techniques 
discussed  previously  were  applied. 
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If  the  control  variables  are  piecewise  constant,  the  state  equations 

can  be  written  as 

x   (k+1)   =  <l     (    (k+l)T,kT)x    (k)    +  T   (    (k+l)T,kT)u    (k) 

"P  P  "P  P  "P  (46) 

x    (k+1)    =   <$      (    (k+l)T,kT)x    (k)    +  r   (    (k+l)T,kT)u    (k) 
~e  e  e  e  ~~ e 

where      $    (    (k+l)T,kT)    and    $    (    (k+l)T,kT)   are  the  respective  state 
P  e 

transition  matrices  [12]  ,  and    T    (  (k+l)T,kT)   ,      T   (  (k+l)T,  kT)    are 


given  by 


(k+l)T 
r   (  (k+l)T,kT)    =  [  <f    (  (k+l)Tft)Bn(t)  dt      (47) 

P  «    i     m  P  P 


P  JkT       P 

(k+l)T 

T   (  (k+l)T,kT)    =  [         <l    (  (k+l)T,t)B    (t)  dt      (48) 
6  J, m       e  e 

kT 

respectively.    Although  the  following  development  can  be  performed  for 

time-varying  linear  systems,  for  notational  simplicity  it  is  assumed  that 

A    (t) ,  A   (t) ,  B    (t) ,  and  B    (t)  are  constant  matrices  in  which  case 
p  e  P  e 

4    (  (k+l)T,kT)    =  4    (T) 

P  P 

4e(  (k+l)T,kT)  =  4e(T) 
Tp(  (k+l)T,kT)  =  r  (T) 
r  (  (k+l)T,kT)  =  r  (T) 
A  discrete  approximation  to  J  can  now  be  written  as 


JD  =    [Xp(N)-Xe(N)]TQ     [x    (N)    -   Xe(N)] 


N-l 

+  T  I       [u    (k)TRD  un(k)  -   ujk)\  ujk)    ]    . 

k=0       P         p_p 
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(49) 


Writing  out  a  few  terms  for  the  pursuer's  system  gives 


x    (1)    =   $    (T)x    (0)   +  r    (T)u    I 
_p  p      _p  p^  /_p 


x    (2)    =   4    (T)x    (1)   +  r    (T)u    (1) 

— p  p      ~p  p      — p 


-   4    (T)  [*    (T)x    (0)    +  r  (T)u    (0)]  +  r    (T)u    (1) 
p  p      -p  P      "~P '  P      ~P 

=   4>s(T)x    (0)  +   <£>    (T)r    (T)u    (0)  +T    (T)u    (1) 

p      — p  p         p      — p  p      —  p 


x    (N)    =   $N     (T)x    (0)    +   £N    X(T)r  (T)u    (0)  +  $N   2(T)r(T)u    (1) 
-p  P  ~P  P  P      -p  P  P      ~p 


+  ...+  *    (T)  T  (T)u    (N-2)    +  T    (T)u    (N-l) 


(50) 


A  similar  expression  can  be  written  for  x    (N) .    Thus  it  is  possible  to 

express  the  final  state  values  in  terms  of  the  initial  state  and  the  N 
values  of  the  control  vector.    Substituting  equation  (50)  into  equation 
(49)  allows  the  payoff  to  be  written  as  a  function  of  the  2Nq  control 


values 


u 
-p 


u    (0) 

u    (1) 
_p 


u 
—e 


u    (N-l) 
~P 


u 
—e 

(0) 

u 
— e 

(1) 

u 
—e 

(N- 

1) 

The  nxNq  matrices  4*       and  4     are  defined  as 

P  e 

—  r     N-l  '       N-2  ' 

4P  s  [ftp    (T)rP(T)  :  4p    (T)rp(T):  -- 
£    =  r^^Vir  (T)  ;  <r^"2(T)r  (t);  -  - 


:rp(T) 


re(T)j  , 


(51) 


(52) 
(53) 
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the  NqxNq  matrices  R  and  R     are  given  by 

P  e 


R     = 
P 


R  0 

P 

0  R 


R      s 
e 


R 


0 

R 


0  0 


0  0      .     .    .    R 


R 


§J 


(54) 


(55) 


and 


9  =  4-N(T)x    (0)    -   <|N(T)x    (0) 

p      — p  e       — e 


(5  6) 


where    9  is  the  "predicted  terminal  miss"  at  time  t=NT  if  no  controls  are 
applied  during  the  preceding  N  intervals  . 
J      can  now  be  written  as 

J^  =  [6+5   u   -5   u   ]T  Q[9+§  u  -<|  u   ]  +  T[u  TR  u     -  u  TR  u    ] 
D  p— p       e— e  p— p     e— e  —  p      p— p      — e      e~e 

=  uT[iTQ<l      +R]u     +     uT[5>TQ^»      -R]u 
— p        pp         P~ P         ~~ e         ee  e^e 


2  -j  uT    feTQ6]-u    [<?TQ4    ]u      -  uT[lTQ6]j  +  6TQ9.    (57) 
.^p        p  — p      p       e    — e       -^e      e  J 


Since  it  was  assumed  that  Q  is  positive  semidefinite  and  R     positive 

P 

definite,  it  follows  that  the  matrix     [<£    Q4      +  R    ]  is  positive  definite 

P         P         P 

and  hence  that  J^  is  a  convex  function  of  u     .    If  J^   is  also  a  concave 
D  — p  D 

function  of  u    ,  then  T      will  have  a  saddle  point.     This  will  be  the  case 
— e  D 
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-  T    - 

if  the  matrix  R     is  weighted  relative  to  <I    Q<l     in  such  a  way  that 
e  e      e 

_  t 

[4   Q4      -R]is  negative  semidefinite  . 
e        e  e 

-  T    - 

Assuming  that    [$    Q<£      -  R   ]  is  negative  semidefinite,  the  pre- 
viously described  methods  can  be  applied.    The  constraints  (44)  and  (45) 
are  written  as 


X    (u   )      = 

~P     P 


I 


q 


u 
"H? 


(-m   )    *0. 


(5  8) 


X    (u  )    = 

— e^ e 


q 


u       -    (-m    )    2:0. 


(59) 


where  I     is  the  q-dimensional  identity  matrix,  m     is  the  2q-dimensional 
q  p 


vector  of  constants  M    ,  m     is  the  2q-dimensional  vector  of  constants 

P 


M    ,  and  the    problem  may  be  stated  in  the  concise  form:    Determine  a 
e 

saddle  point  of  (5  7)  subject  to  the  constraints  (5  8)  and  (59). 

For  (u*,u*)  to  be  a  solution  to  this  problem,  it  is-  necessary  and 
p  — e 

sufficient  that  (u*,u*)  and  the  2  q-dimensional  vectors  u    and  u 
— p  — e  p  e 


satisfy 


7-    T    (u*,u*)    +    [I       !    -I    3  U        =      0 

u    JDnpnB;  q     ;         q      *p 


X    (u*  )     M  =0 

—  p     p        ^  p 


X    (u*) 

-p  ~p 


£     0 


£     0 
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-7-     t   (u*,u*)   +    [i    ;  -I  ] JU 

u        D  — p  -e  q  ;       qJJ=e 


(60) 


K<k)T*< 


X    (u*) 

—  e  — e 


£        0 


The  following  simplified  pursuit-evasion  game  was  solved  by  using 
a  Newton-Raphson  method,  Brown's  algorithm,  and  by  sequential  minimi- 
zation and  maximization  using  the  gradient  projection  algorithm. 


Let 


A  =   A    (t)    =  Ae(t)    = 


B  =  B    (t)   =  B   (t)   = 

p  e 


Q 


0  1 

0  0 

0 

1 


rl 


R 


x    (0) 


r  0 

P 


PJ 


R 


x    (0) 
-e 


e  -> 


10 


-2 
I I 


The  initial  time  was  set  at  0,  the  final  time  at  2,  and  the  sample  interval 
was  taken  as  T=l . 


The  state  transition  matrix  is 

1  T 


®=     S     (T)    =    *e(T)    = 


0 


1 


J 
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and  the  matrix  p    is  given  by 


r  =    r  (t)  =  r  (t)  = 

P  e 


J     becomes 


T3/2 


t       -   ~T 


9/4  3/4 

3/4  1/4 


+ 


-T 
+  u 
— e 


+  2 


u 


1 9/4       3/4 
i  3/4        1/4 

-3 

-1 


_^\ 


-T 
-    u 
"~ p 


r  0 

P 


Pj 

r         0 
e 

0        r 


9/4      3/4 

3/4      1/4 


u 


>  u 

—e 


-T 
u   -  u 
~e     ~e 


~\ 


(61) 


>+   4 


and  for  J^   to  have  a  saddle  point  it  is  sufficient  that 
D 

r        >       0 
P 

r        >       9/4 
e 

Note  that  T       is  a  convex  function  of    u      for  r        >     -   i  but  since  R     was 
D  -P  P  p 

assumed  positive  definite  it  is  necessary  that    r     >  0. 

P 

Values  of  r     =   1.  and  r      =5.    were  initially  chosen  and  with  M     = 
p  e  p 

M      =  2.  ,  the  Newton-Raphson  method,   Brown's  algorithm  and  sequential 
min-max  with  gradient  projection  gave  the  interior  saddle-point 


u    (0)   =    1.000 
P 


u    (0)    =   0.200 
e 


u    (1)   =   0.333 
P 


u    (1)    =   0.067 
e 


The  Newton-Raphson  method  and  Brown's  algorithm  both  converged  in 
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four  iterations  while  the  gradient  projection  method  required  24  minimiza- 
tions and  maximizations  . 

The  value  of  M     was  then  changed  to  M      =  =0  . 8 .    In  six  iterations. 
P  P 

the  sequential  min-max  method  converged  to 

u    (0)    =   0.800  u    (0)    =   0.320 

p  e 

u    (1)    =   0.533  u    (1)    =   0.107 

p  e 

The  Newton-Raphson  method  required  four  iterations  to  converge  to  the 

same  point.     Depending  on  the  starting  point,    Brown's  algorithm  converged 

to  the  unconstrained  solution  or  created  a  singularity  in  the  vicinity  of  the 

constrained  solution,  but  it  did  not  converge  to  the  constrained  solution. 
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VII.     CONCLUSIONS 

Nonlinear  programming  methods  have  been  applied  to  the  solution 
of  constrained  saddle-point  problems,  in  particular,  to  a  type  of  problem 
arising  in  pursuit-evasion  games  with  inequality  constraints  on  the  values 
of  the  control  variables  „    The  payoff  functions  considered  were     real- 
valued,  concave-convex  quadratic  polynomials,  although  nonlinear 
programming  methods  are  not,  in  general,  restricted  to  such  functions. 

Five  techniques  were  suggested  -   -  the  two  most  promising  being 
the  Newton-Raphson  method  to  solve  nonlinear  simultaneous  equations 
obtained  from  the  Kuhn-Tucker  necessary  conditions,  and  the  solution  of 
a  sequence  of  minimization-maximization  problems  using  the  gradient 
projection  algorithm.    The  Newton-Raphson  method  was  found  to  be  some- 
what sensitive  to  the  starting  point  chosen.    Gradient  projection,  while 
insensitive  to  the  starting  point  (even  a  nonfeasible  point  is  acceptable), 
was  found  to  converge  only  when  certain  conditions  on  the  function  were 
satisfied.     These  conditions  were  found  by  an  application  of  the  Contrac- 
tion Mapping  Principle. 

The  primary  advantage  of  these  methods  is  that  they  admit  inequality 
constraints  on  the  variables,  in  fact,  it  is  required  that  the  variables  be 
constrained. 

A  serious  disadvantage  (at  least  in  the  case  of  pursuit-evasion 
problems)  is  that  only  an  open-loop  strategy  is  obtained.  A  pursuit- 
evasion  problem  with  inequality  constraints  for  which  a  closed-loop,  or 
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feedback,  strategy  is  desired  must  (apparently)  be  approached  using 
calculus  of  variations  techniques  [4]. 

In  addition  to  the  inclusion  of  inequality  constraints,  differential 
game  theory  has  many  unanswered  questions.     Perhaps  the  most  obvious 
flaw  in  the  formulation  of  pursuit-evasion  games  is  the  lack  of  realism  in 
the  performance  criterion.     It  has  been  pointed  out  [7]  that  an  aircraft 
commander  being  pursued  by  a  missile  will  be  unsatisfied  with  a  strategy 
which  tells  him  to  conserve  his  energy,  yet  it  was  found  that  for  a  saddle- 
point  to  exist  the  evader's  control  energy  term  must  be  weighted  rather 
heavily  in  comparison  to  the  terminal-miss  term.    This  raises  a  very 
fundamental  question  concerning  what  is  meant  by  "optimal"  in  optimal 
strategies.    In  the  formulation  given  in  this  thesis,  optimal  means  mini- 
max  strategies  which  are  a  saddle  point  of  the  performance  criterion. 
But,  just  as  in  optimal  control  theory,  the  engineer  or  analyst    must  be 
able  to  formulate  realistic  performance  criteria  if  the  "optimal"  control, 
or  strategy,  is  to  have  physical  meaning. 

A  more  realistic  payoff  for  pursuit-evasion  games  would  be  simply 
a  measure  of  the  terminal  miss,  but  a  saddle  point  may  not  exist  for  this 
criterion  and  the  meaning  of  the  term  "solution"  must  be  altered.    Very 
little  information  is  available  on  differential  games  for  which  a  saddle 
point  does  not  exist.     It  is  well  known  that  matrix  games  that  do  not  have 
pure  strategy  solutions  (no  saddle  point  exists)  have  solutions  in  mixed 
strategies.    A  strategy  is  called  "mixed"  if  it  is  chosen  from  a  set  of 
admissible  strategies  in  accordance  with  some  particular  probability 
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distribution.  Berkowitz  [4]  discusses  the  notion  of  mixed  strategies 
for  differential  games  but  observes  that  the  mathematical  difficulties 
which  arise  have  not  been  overcome. 

Another  approach  which  warrants  investigation  for  possible 
application  to  differential  game  theory  is  introduced  by  Salmon  [18]   . 
Although  limiting  his  attention  to  the  design  of  controllers  for  systems 
with  uncertain  parameters,  Salmon  proposes  an  algorithm  to  solve  a 
class  of  minimax  problems  for  which  a  saddle  point  need  not  exist. 

Further  investigation  of  the  methods  suggested  in  this  thesis 
should  include  the  solution  of  meaningful  examples,  e.g.  ,  a  pursuit- 
evasion  problem  with  enough  samples  over  the  time  interval  to  adequately 
represent  the  trajectories  .    It  is  expected  that  the  conditions  for  con- 
vergence of  the  sequential  min-max  method  would  prove  somewhat 
restrictive.     Difficulty  with  convergence  of  Brown's  algorithm  has  already 
been  experienced  and  could  be  expected  with  the  Newton-Raphson  method 
in  larger  dimension  systems  . 
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APPENDIX  A 
In  this  appendix  it  is  shown  that  Theorem  4    of  Rosen's  gradient 
projection  algorithm  [15]    is  equivalent  to  the  Kuhn-Tucker  necessary 
conditions  for  the  nonlinear  programming  problem . 

Kuhn  and  Tucker  stated  the  Maximum  Problem  as 
max  f(x)  ,  x  an  m-vector 

subject  to  X(x)    ^  0.      ,  X   a  k-vector 

x     £  0_      m 
The  necessary  conditions  for  x*  to  be  a  solution  to  this  problem 

are  that  x*  and  some  \i  satisfy: 

k 
7  F(x*)    =  7  f(x*)    +     I    [m    7     X.(x*)]  £    0  (Al) 

x     —  2£   —  .-i1^1-  — 

J7    f(x*)    +  Z     [ii.  7     X.(x*)]}Tx*      =    0  (A2) 

I        X     —  •_-!        1       X       1  —  J    ~ 


x*      >    0_ 

(A3) 

X(x*)     £    0 

(A4) 

X  (x*)T  M     =    0 

(A5) 

Ik     *    £ 

(A6) 

T 
where  F(x)  is  the  Lagrangian  function,  F(x)=f(x)    +  \i_  X(x)  and  the  con- 
straints X(x)s  0_,  x^0_  must  satisfy  the  constraint  qualification[l4]   . 
Rosen  [15  ]  considered  only  linear  constraints  in  Part  I  of  his  algorithm, 
and  linear  constraints  always  satisfy  the  constraint  qualification. 
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The  linear  constraints  can  always  be  written  in  the  form 

X.(x)  =xTn.    -  b.    *     0     ,    i=l ,  ..'.k  (A7) 

1  ~~  ~~i  1 

where  the  n     have  been  normalized,  that  is, 
~i 


i-1, .'-.  j  ,k  .  (A8) 


m 

E    (n..)a 
J-l     " 

=   1 

that 

V    X.(x)  = 

x     i  -/ 

n 
~i 

(A9) 

The  mxk  matrix  Nk  is  defined  as 

Nk  =    [Hi  H2      ...      nk]  (A10) 

(for  linear  constraints  this  corresponds  to  F°  in  [14]),  and  the  k-vector 

b  as 

T 
b   =    [b1#  bS/  .  .  .    ,nk]         .  (All) 

Using  these  definitions  ,  the  system  of  k  inequalities  can  be  written  as 

Nk     2£  "  k     *     0  .  (A12) 

In  addition  Rosen  does  not  require  that  x   £    0_;  therefore,   (Al)  must 
hold  as  an  equality,  and  the  necessary  conditions  become 
7     f(x*)      +      Nk     y.        =        0 

-A. 


Nk      x*    -     b    ^        0_ 


[Nk    x*    -   b]T     ji     =       0 


(A13) 


y,    *      0 
Rosen  has  assumed  that  the  k  constraints  form  a  convex,  bounded 
region  R.     Thus,  it  must  be  that  k^m+1  .     Clearly,  at  most  m  of  the  k 
hyperplanes    X(x*)    =   0_    are  linearly  independent.     Letn.    ,  i=l ,  2,...,q, 
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be  a  set  of  linearly  independent  unit  vectors  which  define  a  set  of  q 
linearly  independent  hyperplanes 

X.(x*)    =   0         ,      i=l,2, q      ;    l<q<m  .  (A14) 

If    X  .  (x*)    >  0    for  all  i ,  i=l ,  .  .  . ,  k    ,  then  ^   must  equal  0_.     In  this 

case  x*  is  an  interior  point  of  the  region  R  and  the  necessary  conditions 

reduce  to 

7    f(x*)    =      0  ,  (A15) 

x  v—  — 

the  familiar  necessary  condition  for  an  unconstrained  problem. 

Now  assuming  that  X  .  (x*)    =   0  ,  i=l ,  .  .  .    ,q  ,  which  can  be  the  case 
for  at  most  m  of  the  k  hyperplanes  ,  the  mxq  matrix 

Nq   =    [&]_    n2   ...  n    ]  (A16) 

can  be  formed  from  the  vectors  defining  the  above  hyperplanes.    Also, 
letting     ^x     be  the  vector  of  the  jli  .  which  correspond  to  X  .  (x*)   =   0,  the 
Kuhn-Tucker  necessary  conditions  can  be  written  as 

7xf(x*)    +  NqMq      =0  (A17) 

X.(x*)  =0       ,     i=l, q  (A18) 

i  — 

Mq     *    £  .  (A19) 

Using  the  psuedo-inverse  of  N     to  find  £     from    (A17)  gives 

u      =    (NT   N   )_1    NT      [-7      f(x*)]  .  (A20) 

cq  q       a  q  x      — 

From  (A19)  this  must  be    ^  0_   ,    or 

(NT     N   )_1    NT    [7    f(x*)    ]      s   0         .  (A21) 

q         q  q  x    —  — 
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Equation  (A17)  can  now  be  written  as 

7  f(x*)    +  N    (NT   N   )_1    NT    [-7   f(x*)  ]  -      0 
x  y—  '  q      q       q  q    u     x  x—  — 

[  I   -   N    (NT   N   J"1    NT    ]   7    f(x*)      =      0  .  (A22) 

qqq  q         x  —  —  •  v/ 

Using  Rosen's  definition  of  the  projection  matrix 

P       =     I   -   N    (NT   N   J"1  NT  ,  (A23) 

q  q      q      q  q 

equation  (A22)  becomes 

P      [7f(x*)   ]     =      0  .  (A24) 

q  x 

Equations   (A21)  and  (A24)  are  precisely  the  necessary  conditions 
given  by  Rosen  [15]   in  Theorem  4   for  x*  to  be  a  constrained  maximum  of 
f (x)  .     If  f(x)  is  a  concave  function  of  x  the  Kuhn-Tucker  conditions,  and 
thus  equations  (A21)  and  (A24),  are  also  sufficient  f or  x*  to  be  a  con- 
strained global  maximum  of  f  (x) . 
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APPENDIX  B 

The  Contraction  Mapping  Principle  [13]  can  be  applied  to  find 
sufficient  conditions  for  a  sequence  of  points  obtained  from  the  sequential 
minimization-maximization  of  a  real-valued,  concave-convex  function 
with  continuous  and    bounded  second  partial  derivatives  to  converge  to 
the  saddle  point. 

A  mapping  A  of  the  space  R  into  itself  is  said  to  be  a  contraction 
if  there  exists  a  number  &  <  1    such  that 

p  (Ax,AyJ  <  ap(x,yj 
for  any  two  points  x,y_  cR,  where  R  is  an  arbitrary  metric  space,  and    p 
denotes  the  choice  of  the  distance  in  R.     In  this  thesis  p  is  defined  as 

p(x,vj    =   {L    (x.    -   y  )3}*     =   {  [  x-y_]  T  [x-y_] T  }*      . 
i=l      1  i      J 

The  functions  examined  in  this  thesis  may  all  be  represented  by 

quadratic  polynomials  of  the  form 

rn  rp  m  rp  rp 

f(£'X)    =  2£  A*     +  Y_B.Z  +  2££  +  x_  DY_  +  X  ®   +  9      •  (B1) 

This  can  be  reduced  by  linear  transformations  to  a  simpler  form  by  intro- 
ducing new  vectors  x  and  y_  defined  as 

x  =  x  +  h 

(B2) 
y_  =  y_  +  k 

where  h  and  k.  are  constant  vectors  to  be  determined.     Substituting  (B2) 

into  (Bl)  gives 

f(x,yj    =    (x+h)TA(x+h)      +      (y_+k}T  B(y+k)      +      (x+h)Tc 

+      (x+h)T  D(y_+k)      +      (y_+k)Te   +  g  (B3) 
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which  can  be  written  as 


T  T  T  T 

f(x,y}    =  x    Ax  +  y_    By   +  x    Dy_  +  x_  (2Ah   +  c   +  Dk) 


+  Y.  (2Bk  +  e   +  Dh) 


rp  rp  rp  rp  m 

+   (h    Ah  +  k    Bk   +  he  +  h    Dk   +  k  e   +  g) 


(B4) 


T  T 

By  setting  the  coefficients  of  x     and  y_     equal  to  zero,     the  values  of  h 

and  k_  are  determined  from 

2Ah   +  Dk   +  c  =  0 


Dh  +  2Bk   +  e   =  0 


or, 


2A       D 


D      2B 


If  the  matrix 


2A       D 
D      2B 


(B5) 


has  an  inverse,   (B5)  can  be  solved  for  h  and  k. 


This  matrix  will  have  an  inverse  if  A  is  positive  definite  and  B  is  negative 
definite.     Equation  (B4)  then  has  the  form 


T  T  T 


(B6) 


where  A  is  positive  definite,  B  is  negative  definite,  K  is  a  constant,  and 
f  has  a  saddle  point  at  the  origin. 

Allowing  the  variables  x  and  y_  to  be  unconstrained,  for  a  particular 
Y_  the  function  f  is  minimized  when   7  f (x,yj  =0  .    The  value  of  x  at  this 


point  is 


7  f(x,yj    =   2Ax   +  Dy_  =   0. 


x  =  -J  A"1  Dy_ 


(B7) 
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Similarly,  for  any  x,  the  function  f  is  maximized  at 
7  f(x,_yj   =  2By_  +  Dx  =   0. 

Y_  =   -J  B"1  D&  .  (B8) 

Let  xJr.1  ,Ys   be  an  arbitrary  point.    Minimizing  f  with  respect  to  x,  the 
next  value  of  x    (x  i  )  will  be 

x,   =  -4A_1DXi  •  (B9) 

Maximizing  f(xt/Y)  with  respect  to  y_  gives  as  the  new  value  of  y_ 


Zi+i   =   "4  B^DXj 

=     iB^DA^D^i 


Minimizing  f(x,Xi+i)  with  respect  to  x  obtains 

2.i+i   =   "4  A_1DZ1+i 
=     i  A^DB""1  Dx , 

Equations  (BIO)  and  (Bll)  can  be  written 


JLi+i    ~  GiLi 


(BIO) 


(BID 


(B12) 


where 


z.i   = 


*i 


Zi 


and 


C   = 


iA^DB^D 


0 


i  B_1DA_1  D 


^^.l  +  i  is  closer  to  the  origin  (the  saddle  point)  than  z^   ,  i.e.,  if 

2n 

i-  i 


15.1+1  II  "^  II  ^.i II   where  ||  z_t 


U.j  )2 


,  the  sequential  min- 


max  procedure  will  converge.     This  will  be  the  case  if  (B12)  is  a 
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contraction  mapping.     From  page  45  [13],  a  sufficient  condition  for  the 
mapping  C  to  be  a  contraction  is 

Lt   Sj    at  ja  £    a  <  1  (B13) 

where  the  a%  i   are  elements  of  C  . 

An  alternative  result  may  be  derived  from  the  definition  of  a  con- 
traction mapping.    A  mapping 

z   =   Cz. 
is  a  contraction  if 

o(Cz)    <ap(z)  (B14) 

where  a  <  1    ,  that  is  , 

T    T  T 

z  C  Cz   <    a    z  z  (B15) 

Thus,  if 

T     T  T 

z  C   Cz     <     z   z  (B16) 

then  the  mapping  C  is  a  contraction.    Conversely  if 

T    T  T 

z   C   Cz      >     z    z  (B17) 

then  C  is  not    a  contraction  mapping. 

It  is  well-known  that  on  the  sphere    ||  z_||   =1  ,  the  maximum  value  of 

T    T 
£  C   C_z   is 

max   zTCTCz      =      X,  (B18) 

z_—  — 

T 
where   Xi   is  the  largest  eigenvalue  of  C   C ,  and  that  the  minimum  value 


is 


min   zTC   Cz   =  X  AT  (B19) 

z—  —  N 
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T  T 

where  \      is  the  smallest  eigenvalue  of  C   C .     The  matrix  C   C  is  real, 

symmetric  and  positive  definite,  hence  its  eigenvalues  are  all  real  and 
positive . 

From  (B18)  and  (B19) 

XN    l|z||S   *   zTCTCz      <     Xi    ||z||3  .  (B20) 

Looking  at  the  right-hand  inequality  first 

zTCTCz      <     Xx    ||  z  ||3  ,  (B21) 

if  X  i  <  1    then  it  follows  that 

zTCTCz     <     zTz      =      ||  z  IIs  (B22) 

and  the  mapping  C  is  a  contraction. 
From  the  left-hand  inequality 

0|z||     *     zTCTCz  (B23) 

and  if     \       >  1  , 

T    T  T 

z   C   Cz      >     z_  z  (B24) 

and  C  is  not  a  contraction  mapping. 

Thus,  Xi  <1    is  a  sufficient  condition  for  C  to  be  a  contraction 

mapping  and     X      >  1  is  a  sufficient  condition  for  C  not  to  be  a  contraction 

N 

mapping. 
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